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ABSTRACT 

In self-adapted testing (SAT), examinees select the 
difficulty level of items administered. This study investigated three 
variations of prior information provided when taking an SAT: (1) no 
information (examinees selected item difficulty levels without prior 
information); (2) view (examinees inspected a typical item from each 
difficulty level prior to taking the SAT); and (3) route (examinees 
answered a typical item from each difficulty level and were informed 
prior to the SAT of the level that best matched their performance). 
Subjects were 218 students in an introductory statistics course, 85 
percent of whom were undergraduates. No significant differences in 
test performance or anxiety were found as a function of providing 
examinees more information. Four tables are included, (Contains 9 
references, ) (Author/SLD) 
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Effects of Informed Item Selection on Test Performance and Anxiety for 
Examinees Administered a Self-Adapted Test 

(Abstract) 



In self-adapted testing (SAT) examinees select difficulty levels of items 
administered. This study investigated three variations of prior information 
provided when taking a SAT, (1) No Information: examinees selected item 
difficulty levels without prior information; (2) View: examinees inspected a 
typical item from each difficulty level prior to the SAT; (3) Route: examinees 
answered a typical item from each difficulty level and were informed prior to the 
SAT of the level that best matched their performance. No significant difference 
in test performance or anxiety was found as a function of providing examinees 
more information. 
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Effects of Informed Item Selection on Test Performance and Anxiety for 
Examinees Administered a Self-Adapted Test 

Introduction 

When an examinee is administered a typical computer adaptive test 
(CAT), an item selection algorithm is used to sequentially select items to 
administer to each examinee in Cxder to tailor the test difficulty to 'che ability 
level of the examinee. One variant of CAT is self-adapted testing (SAT; Rocklin 
& O'Donnell, 1987). An examinee administered a SAT sequentially selects the 
level of difficulty of items to be administered, therefore personally tailoring the 
test to his or her own perceived ability level. 

Under the item invariance property of item response theory (IRT), ability 
estimates should not be influenced by the difficulty of the items administered. 
Recent research comparing SAT to CAT, however, has found significantly higher 
ability estimates for examinees administered a SAT (Rocklin and O'Donnell, 
1991; 'Roos, Plake and Wise, 1992; Wise, Plake, Johnson, and Roos, 1992; Vispoel 
and Coffman, in press). 

The primary distinguishing feature between CAT and SAT is the 
involvement of the examinee in the item selection process in SAT. In the typical 
SAT administration, examinees are informed of relative difficulty levels of the 
items. Usually the distribution of item difficulties is broken into six or eight 
levels. The examinees may or may not be shown a typical item in the middle 
level of difficulty. Therefore, based on little (typical item from the middle 
difficulty level) or no information, examinees are asked to select the difficulty 
level for their first item to be administered. Research into the difficulty level 
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selection patterns for examinees administered a SAT under these conditions has 
shown little pattern of coherence (Johnson, Roos, Wise, and Plake, 1991). 

The purpose of this study was to focus on this dimension of SAT: 
examinee selection of difficulty level. Three conditions of information potentially 
useful to examinees in making item difficulty selections were used: 



Condition 1, No Information: Examinees were asked to sequentially 
select the difficulty level of the items to be administered with no prior 
information. 



Condition 2, View: Examinees were shown a typical item from each of the 
six difficulty levels prior to being administered the SAT. 

Condition 3, Route: Examinees were administered a typical item from 
each of the six difficulty levels and informed, as determined by a 
maximum likelihood estimate of their proficiency based on the six-item 
mini-test taken prior to the SAT, which level best matched their 
proficiency. 

The following research questions were addressed in this study: 

1. Does having more information about the test items affect examinee 
performance on the SAT? 

2. Does having more information about the test items affect examinee anxiety 
when taking a SAT? 
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3. Does having more information about the test items affect examinee item 
difficulty level selection for the first item to be administered in the SAT? 

Method 

Subjects 

Examinees were students in multiple sections of an Introduction to 
Statistics course in the Spring, 1993 semester at a large midwestern university. 
Students were required to take a Statistics Readiness Examination during the first 
week of class. Those students who score below a pre-determined cutoff were 
required to attend a one-hour mathematics review session the second week of 
class. 

Instruments 

Statistics Readiness Test, Students were administered 20 items from a 91 
item pool. These items were designed to measure basic algebra and mathematics 
reasoning necessary for successful completion of an introductory statistics class. 
The iteirts were calibrated using a modified 1 -parameter IRT model More 
information about the item pool is presented in Wise et al. (1992). The SAT was 
administered using MicroCAT™ software (Assessment Systems Corporation, 
1988). 

Anxiety Instruments . Two instruments were administered to assess 
anxiety: The Revised Mathematics Anxiety Rating Scale (RMARS; Plake and 
Parker, 1982) and State scale of the State-Trait Anxiety Inventory (STAI; 
Spielberger, Gorsuch, and Lushene, 1970). 
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Background Questionnaire . A one-page questionnaire was administered 
that gathered student information regarding number of previous algebra classes 
and how recent that algebra coursework had been. The questionnaire also 
included questions pertaining to gender and age. 

Procedure 

During the first day of class, students were informed of the testing 
requirement and asked to sign up for a testing time during the first week of 
classes. Also, during the first day of class, students filled out the background 
questionnaire and the RMARS. 

Students were tested in a room containing 12 IBM PS/2 Model 55 
microcomputers. Experimental conditions were randomly assigned to the 12 
computers at the beginning of each of the four days of testing. When a student 
arrived at the testing center, a monitor greeted the students and allowed the 
student to select one of the available machines for testing, 

Prior to beginning the test, the examinee was administered the State 
Anxiety scale of the STAI. Depending on the experimental condition, some 
examinees were given information about the difficulty of the items in the 
difficulty levels (View or Route conditions). For the Route condition, 
performance on the six practice items was not scored as part of the 20-item 
operational test. No time limit was imposed but most students finished the 20- 
item test in 20 - 25 minutes. 

All students were given basic directions and informed that the items in the 
test fell into six difficulty levels, ranging from very easy to very hard. Before 
each item was administered, the student was instructed to select the difficulty 
lev J of their next item from the six levels. The items were randomly arranged 
within each difficulty level. If no items were remaining in the level selected 
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(there were 13 - 14 items available in each difficulty level at the start of the test 
administration), the student was informed that no items were available in that 
level and instructed to select another level. After answering each item, the 
student was informed of the correct answer for the item. This sequence of item 
difficulty level selection and student response continued until a total of 20 items 
was administered. 

Upon completion of the test, the student was instructed to contact the 
monitor. The student was then again administered the State Anxiety scale of the 
STAI. Next, the student was informed whether or not he or she was required to 
attend the mathematics review session. The students were then thanked for their 
participation and dismissed. 

Results 

A total of 218 students participated in the study, of which 85% were 
undergraduates and 15% graduate students. There were 92 males and 126 
females. 

Research Question 1: Does having more information about the test items 
affect examinee performance on the SAT? 

Examinee performance on the Statistics Readiness Test is summarized in 
Table 1. An analysis of covariance, with RMARS as the covariate, was run to 
investigate differential test performance for the three conditions. A non- 
significant main effect for condition was found, F(2, 204) = 0.13, p=.88. 
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Research Question 2: Does having more information about the test items 
affect examinee anxiety when taking a SAT? 

Examinee anxiety was investigated by a repeated measures analysis of 
variance, using pre and post measures from the State scale of the STAI. Means 
and standard deviations for post STAI are displayed in Table 2. A non- 
significant effect for condition was found, F(2 / 204) = 0.21, p<.81. 

Research Question 3. Does having more information about the test items 
affect examinee choice of the difficulty level selection for the first item to be 
administered in the SAT? 

Tables 3 and 4 summarize examinee choice of the difficulty level for the 
first item to be administered by the SAT. Table 3 shows the distribution of 
difficulty level selected for their first item by examinees in the three conditions. 
Table 4 shows the relationship between the proficiency level of examinees in the 
Route condition (as determined by a maximum likelihood estimate of their 
proficiency based on the six-item mini-test taken prior to the SAT) and their 
selection of difficulty level for the first SAT item. 

A Chi-Square Test of Independence of choice of difficulty level for first 
SAT item by condition was significant (X 2 (10) = 28.83, jj < .01)). Pairwise follow- 
up tests revealed a non-significant relationship between choice of difficulty of 
first SAT item and condition for the ' and Route conditions (X 2 (5) = 8.59, £ < 
.15). However, significant effects were found for the comparison between the No 
Information condition and each of the other conditions (View: X^(5) = 18.79, p. < 
.01; Route: X 2 (5) = 15.49, g < -01)). Examinees in the View and Route conditions 
had a tendency to choose higher difficulty categories for their first SAT item than 
did examinees in the No Information condition; percentages of examinees 
selecting difficulty levels 4, 5 or 6 for the first SAT item were 47% for No 
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Information condition, 67% for the View condition, and 70% for the Route 
condition. 

Further, examinees in the Route condition appear to have been influenced 
by the information provided to them about the difficulty level that best matched 
their proficiency: 78% of the candidates selected as the difficulty level for their 
first item the same difficulty level identified as the one that best matched their 
proficiency level. Therefore, the level of difficulty selected for their first SAT 
item by these examinees appears to have been affected by the information 
provided to them by the short mini-test. This appeared to be good advice as 75% 
of these examinees, across the six difficulty levels, correctly answered the first 
item they were administered. Examinees in the View (80%) and No Information 
(72%) conditions also showed high succes in answering the first item in the SAT 
correctly. However, because examinees in the No Information condition were 
more likely to select items from the lower difficulty levels for their first SAT item, 
this high level of success could be a function of the tendency to select easier 
difficulty categories for their first SAT item. 

Discussion 

Examinee performance and anxiety did not appear to be influenced by 
having more information about the difficulty levels or by additional information 
about the match of the examinee proficiency to item difficulty level. It could be 
that these students, as advanced undergraduate and graduate students, were 
sufficiently self-knowledgeable of ability in this area to not require additional 
information for making item difficulty level selections. 

The choice of the first difficulty level for the SAT, though, was influenced 
by information provided to the examinees about the difficulty level that was the 
best match to their proficiency level. While this appeared to be good advice, as 
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the examinees tended to perform well on the item administered as their first test 
question, the effect of the information about their proficiency level and the 
difficulty levels of the items did not have u noticeable effect on overall 
proficiency estimates for the students in the Route condition as compared to that 
of students in the other information conditions. It is interesting to note, however, 
that the students in the Route condition showed the highest overall estimation of 
ability and also the highest level of post state anxiety. The higher scores, 
however, were not significantly different from those for examinees in the other 
two conditions. 

Most of the research in SAT has been done on college-aged students. It is 
possible that these students are self-knowledgeable of their abilities and therefore 
do not make different item difficulty selection decisions when given additional 
relevant information. The generalizability of the SAT research to less 
sophisticated students needs to be addressed. 
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Table 1 

Means and Standard Deviations of Estimated Ability (Theta) Broken Down by 
Information Condition 



Condition 

No Information View Route 

N 68 71 79 

Mean 0.438 0.448 0.586 

sd 1.300 1.127 1.280 
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Table 2 

Means and Standard Devia ' : ^ns of Post Anxiety Broken Down by Information 
Condition 



Condition 

No Information View Route 

N 68 71 79 
Mean 38.82 38.00 39.43 
id 1L2Z 1L45 12.33 
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Table 3 

Number of Examinees by Condition Selecting Each of the Six Difficulty Levels 
for their First Item SAT 



Condition 


Difficulty Level of First Item Selected 
1 2 3 4 5 6 




No Information 


6 


5 


25 


16 


6 


10 


68 


View 


2 


13 


9 


18 


16 


13 


71 


Route 


6 


6 


12 


14 


17 


24 


79 


TOTAL 


14 


24 


46 


48 


39 


47 


218 
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Table 4 

Number of Examinees by Proficiency Level in the Route Condition Selecting 
Each of the Difficulty Categories as Their First SAT Item 



Difficulty Level Selected for First Item in SAT 



Proficiencv Level 


1 


2 


3 


4 


5 


6 


Total 


1 


3 


1 










4 


2 




3 


1 








4 


3 


1 


1 


6 








8 


4 




1 


3 


8 






12 


5 


2 




1 


3 


13 


4 


23 


6 






1 


3 


4 


20 


28 


TOTAL 


6 


6 


12 


14 


17 


24 


79 
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